String Data in Python

The Python language actually has two different kinds of strings: one for storing text, and one for storing raw bytes. A text string stores data internally as Unicode, whereas a byte string stores raw bytes and displays ASCII (for example, when sent to print).

Adding to the confusion, Python 2 and Python 3 use different (but overlapping) names for their text strings and byte strings. The Python 3 terminology makes more sense, so you should learn it and then translate to Python 2 when working there.

>>> text_str = 'The quick brown fox jumped over the lazy dogs.'
>>> type(text_str)
<class 'str'>

>>> byte_str = b'The quick brown fox jumped over the lazy dogs.'
>>> type(byte_str)
<class 'bytes'>

>>> text_str.encode('utf-8')
b'The quick brown fox jumped over the lazy dogs.'

>>> 'foo' == b'foo'
False
>>>
>>> d = {'foo': 'bar'}
>>> d[b'foo']
Traceback (most recent call last):
  File "stdin", line 1, in <module>
KeyError: b'foo'

>>> 'foo' + b'bar'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't convert 'bytes' object to str implicitly

>>> b'foo %s' % 'bar'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for %: 'bytes' and 'str'

>>> 'foo %s' % b'bar'
"foo b'bar'"

>>> byte_str = 'The quick brown fox jumped over the lazy dogs.'
>>> type(byte_str)
<type 'str'>

>>> text_str = u'The quick brown fox jumped over the lazy dogs.'
>>> type(text_str)
<type 'unicode'>

>>> 'foo' + u'bar'
u'foobar'

>>> import sys
>>> sys.getdefaultencoding()
'ascii'

>>> 'foo' == u'foo'
True
>>>
>>> d = {u'foo': u'bar'}
>>> d['foo']
u'bar'

>>> text_str = u'\u03b1 is for alpha.'
>>>
>>> text_str.encode('utf-8')
'\xce\xb1 is for alpha.'
>>>
>>> text_str.encode('utf-8').encode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 0:
     ordinal not in range(128)

text_str.encode('utf-8').decode('ascii').encode('utf-8')

>>> from __future__ import unicode_literals
>>> text_str = 'The quick brown fox jumped over the lazy dogs.'
>>> type(text_str)
<type 'unicode'>
>>> bytes_str = b'The quick brown fox jumped over the lazy dogs.'
>>> type(bytes_str)
<type 'str'>

Observing the Difference

Consider a text string that contains non-ASCII characters, such as the text string in the following code, which says “Hello, world” Google-translated into Greek (note that this is Python 3 code):

>>> text_str = 'Γεια σας, τον κόσμο'
>>> type(text_str)
<class 'str'>

The first thing to note about this text string is that it cannot be encoded to a bytes instance using the ascii codec at all.

>>> text_str.encode('ascii')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3:
     ordinal not in range(128)

This is because ASCII does not have Greek characters, so the ASCII codec does not have any way to translate them into raw byte data. This is fine, though, because that is what the utf-8 codec is for, as shown here:

>>> text_str.encode('utf-8')
b'\xce\x93\xce\xb5\xce\xb9\xce\xb1 \xcf\x83\xce\xb1\xcf\x82,
\xcf\x84\xce\xbf\xce\xbd \xce\xba\xcf\x8c\xcf\x83\xce\xbc\xce\xbf.'

Several things are worth noting at this point. First and foremost, this is the first string you have encountered where the text string and the byte string look substantially different. The repr of the text string looks like human-readable Greek, whereas the repr of the byte string looks like it is intended to be machine-readable.

Also, notice that the lengths of the strings are actually not the same.

>>> byte_str = text_str.encode('utf-8')
>>> len(text_str)
20
>>> len(byte_str)
35

Why is this? Remember the problem that Unicode exists to solve: ASCII assumes a 1:1 correlation between bytes and characters, which puts a substantial limitation on the number of characters available.

Unicode allows for many more characters to exist by breaking out of this limitation. UTF-8 characters are variable length. A single Unicode character may be as small as a single byte (for the characters on the ASCII table), or as large as 4 bytes.

In the case of the example Greek text, most characters are 2 bytes, which is why the len of the byte string is almost double the len of the text string. However, the spaces, period, and comma (visible in the byte string as such) are all ASCII characters, and only take 1 byte each.

Unicode Is a Superset of ASCII

Why do the text strings and byte strings that only contain ASCII characters look so similar when printed, but the Unicode strings look so different?

By convention, you print the bytes in the ASCII range as their ASCII characters. Additionally, Unicode is structured in such a way as to make it an exact superset of ASCII. This means that the characters in the Latin alphabet, as well as the common punctuation symbols, are represented the same way in Unicode strings as well as byte strings.

This has another important meaning. Any valid ASCII text is also valid Unicode text.

Encodings Are Not Cross-Compatible

It is important to recognize that while many encodings are structured as supersets of ASCII, they are often not compatible with one another. Outside of ASCII, there is little or no overlap between the latin-1 and utf-8 codecs.

Consider the difference in byte strings encoded using each codec.

>>> text_str = 'El zorro marrón rápido saltó por encima ' + \
...            'de los perros vagos.'
>>> text_str.encode('utf-8')
b'El zorro marr\xc3\xb3n r\xc3\xa1pido salt\xc3\xb3 por encima de los
     perros vagos.'
>>> text_str.encode('latin-1')
b'El zorro marr\xf3n r\xe1pido salt\xf3 por encima de los perros vagos.'

Because of this, a string encoded using one codec is unable to be decoded using the other codec. If you try to take a byte string representing text encoded using latin-1 and decode it as utf-8, the Unicode codec will realize that it is encountering an invalid character sequence and fail.

>>> text_str.encode('latin-1').decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 13:
     invalid continuation byte

Worse, if you try to take a byte string representing text encoded with utf-8 and decode it as latin-1, the (more permissive) codec will successfully return a text string, but with garbled text.

>>> text_str.encode('utf-8').decode('latin-1')
'El zorro marrón rápido saltó por encima de los perros vagos.'

It is impossible to infer based on the content of a byte string what encoding is in use. However, many common document formats and data-transfer protocols provide a mechanism to declare what encoding is in use. On the other hand, it is also possible that a document will incorrectly specify its character encoding.

Python 3

In Python 3, files are ordinarily decoded automatically for you. Consider the following file with Unicode text, encoded using UTF-8:

Hello, world.
Γεια σας, τον κόσμο.

Opening and reading this file in Python 3 gives you a text string (not a byte string).

>>> with open('unicode.txt', 'r') as f:
...   text_str = f.read()
...
>>> type(text_str)
<class 'str'>

This code example is making a few critical assumptions that are important to understand.

The biggest assumption being made is how to decode the file. Text files do not declare how they are encoded. There is no way for the interpreter to know whether it is getting UTF-8 text, Latin-1 text, or something else entirely.

Python 3 decides which encoding should be used based on what kind of system it is running on. A function is available to expose this: locale.getpreferredencoding(). On Mac OS X and on most Linux systems, the preferred encoding is UTF-8.

>>> import locale
>>> locale.getpreferredencoding()
'UTF-8'

However, most Windows systems use a different encoding called Windows-1252 or CP-1252 to encode text files, and running the same code in Python 3 on Windows reflects this.

>>> import locale
>>> locale.getpreferredencoding()
'cp1252'

It is important to note explicitly that the preferred encoding that locale.getpreferredencoding() provides is based on how the underlying system operates. It is reflective, not prescriptive. A text file with special characters saved on almost any system (using that system's default tools) and then opened using open in Python 3 will probably be decoded correctly.

However, files are not opened solely on the same type of system on which they are created. This is where the assumption becomes problematic.

>>> with open('unicode.txt', 'r', encoding='utf-8') as f:
...   text_str = f.read()
...
>>> type(text_str)
<class 'str'>

>>> text_str
'Hello, world. \nΓεια σας, τον κόσμο.\n'

>>> with open('unicode.txt', 'rb') as f:
...   byte_str = f.read()
...
>>> type(byte_str)
<class 'bytes'>

>>> byte_str
b'Hello, world.\n\xce\x93\xce\xb5\xce\xb9\xce\xb1 \xcf\x83\xce\xb1\xcf\x82,
     \xcf\x84\xce\xbf\xce\xbd \xce\xba\xcf\x8c\xcf\x83\xce\xbc\xce\xbf.\n'

>>> byte_str.decode('utf-8')
'Hello, world. \nΓεια σας, τον κόσμο.\n'

Python 2

In Python 2, the read method will always return a byte string, regardless of how the file was opened.

>>> with open('unicode.txt', 'r') as f:
...     byte_str = f.read()
...
>>> type(byte_str)
<type 'str'>

Note that the b modifier was not used in the second argument to open, but a str instance (which is a byte string in Python 2) was returned anyway.

You can get a text string by using decode, just like on a byte string that comes from any other source.

>>> byte_str
'Hello, world.\n\xce\x93\xce\xb5\xce\xb9\xce\xb1 \xcf\x83\xce\xb1\xcf\x82,
     \xcf\x84\xce\xbf\xce\xbd \xce\xba\xcf\x8c\xcf\x83\xce\xbc\xce\xbf.\n'
>>>
>>> byte_str.decode('utf-8')
u'Hello, world.\n\u0393\u03b5\u03b9\u03b1 \u03c3\u03b1\u03c2,
     \u03c4\u03bf\u03bd \u03ba\u03cc\u03c3\u03bc\u03bf.\n'

Because Python 2 always provides byte strings, the open function does not have an encoding keyword argument, and attempting to provide one will raise TypeError.

If you are writing code that is intended to be run on Python 2, the best and safest way to do so is to always open files in binary mode (using b) and, if you are expecting textual data, decode it yourself.

Suppressing Errors

Sometimes when you are decoding or encoding text using strict codecs (such as utf-8 or ascii), you do not want a strict exception when the codec encounters text that it does not know how to handle.

The encode and decode methods provide a mechanism to ask a codec to behave differently when it encounters a set of characters that it cannot handle. Both methods take an optional second argument, errors, specified as a string. The default value is strict, which is what raises exception classes such as UnicodeDecodeError. The two other common error handlers are ignore and replace.

The ignore error handler simply skips over any bytes that the codec does not know how to decode. Consider what happens if you attempt to decode your Greek text as ASCII, as shown here:

>>> text_str = 'Γεια σας, τον κόσμο.'
>>> byte_str = text_str.encode('utf-8')
>>> byte_str.decode('ascii', 'ignore')
' ,  .'

The ASCII code does not know how to handle any of the Greek characters, but it does know how to handle the spaces and punctuation. Therefore, it preserves those, but strips all of the foreign characters.

The replace error handler is similar, but instead of skipping over unrecognized characters, it replaces them with a placeholder character. The exact placeholder character varies slightly based on the situation (whether encoding or decoding, and what codec is in use), but is usually either a question mark (?) or a special Unicode question mark diamond character ().

Here is the result if you try to decode your Greek text using the ascii codec and the replace error handler:

>>> text_str = 'Γεια σας, τον κόσμο.'
>>> byte_str = text_str.encode('utf-8')
>>> byte_str.decode('ascii', 'replace')
' ,  .'

And here is the result if you try to encode your Greek text to a byte string using the ascii codec and the replace error handler:

>>> text_str = 'Γεια σας, τον κόσμο.'
>>> text_str.encode('ascii', 'replace')
b'???? ???, ??? ?????.'

You may notice that when using the replace error handler, the number of replacement characters may not be 1:1 with the number of characters in the actual text string. When decoding a byte string using the ascii codec, the codec has no way of knowing how many bytes correspond to each character, so it ends up showing more question marks than there are actual characters in the text string.

Registering Error Handlers

It is possible to register additional error handlers if the built-in ones are insufficient. The codecs module (where the default error handlers are defined) exposes a function for registering additional error handlers, named register_error. It takes two arguments: the name for the error handler and the actual function that does the error handling.

That function receives the exception that would otherwise be raised, and is responsible for re-raising it, raising another exception, or returning an appropriate string value to be substituted into the resulting string.

The exception instance contains start and end attributes that correspond to the substring that the codec is unable to encode or decode. It also has a reason attribute with a human-readable explanation of the reason why it is unable to encode or decode the characters in question, and an object attribute with the original string.

xIf returning a replacement value, the error function must return a tuple with two elements. The first element is the replacement character or characters, and the second is the position in the original string where encoding or decoding should continue. Usually, this corresponds to the end attribute on the exception instance. If you do this, be careful with the start position you return. It is very easy to get into an infinite loop scenario.

The following example simply replaces characters with a different substitution character:

import codecs

def replace_with_underscore(err):
    length = err.end - err.start
    return ('_' * length, err.end)
codecs.register_error('replace_with_underscore', replace_with_underscore)

This error handler replaces unknown characters, but using underscores rather than question marks. The following is what happens if you decode a byte string with Unicode Greek text using the ascii codec and this error handler:

>>> text_str = 'Γεια σας, τον κόσμο.'
>>> byte_str = text_str.encode('utf-8')
>>> byte_str.decode('ascii', 'replace_with_underscore')
'________ ______, _____ __________.'